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ABSTRACT 



The acquisition of factual knowledge is increasingly 
insufficient to prepare students to cope in this complex and rapidly changing 
world. Students need to learn effective ways of thinking so they can use 
their acquired knowledge flexibly, solve diverse problems, and become good 
independent learners. Two pervasive educational problems limit the possible 
efficacy of any instruction: (1) lack of individual guidance and feedback; 

and (2) deficiencies of basic cognitive functions. A computer- implemented 
reciprocal- teaching strategy can help students learn important cognitive 
abilities. A prototype computer tutorial was designed to teach college 
students Newton’s second law of motion, a principle taught in every basic 
physics course and one with which students often have difficulty. In the 
basic implementation tutorials, PAL (the computer used as Personal Assistant 
for Learning) plays the role of a coach who makes decisions according to the 
methods suggested by the student, and then assesses and corrects the 
students' implementations. In the basic coaching tutorials, the roles of 
implementer and coach are reversed, with the student practicing how to make 
appropriate decisions and assessing PAL's implementation. The tutorial 
programs were tested in a pilot experiment in an introductory physics course 
for college science majors. Student volunteers (n=45) were divided into a PAL 
group, a tutoring group, and a control group for a homework assignment 
dealing with the application of Newton’s laws. Students in the PAL and 
tutoring groups performed appreciably better than students in the control 
group. Students had very positive reactions to the PAL tutorials, and seem to 
have acquired a more explicit knowledge about relevant thinking processes 
from using the tutorials. Implications are discussed. (Contains 14 
references . ) (SWC) 
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Educational challenge of our technological age . The mere acquisition of factual knowledge is 
increasingly insufficient to prepare students to cope with our complex and rapidly changing world. Instead, 
they need to learn effective ways of thinking so that they can use their acquired knowledge flexibly, solve 
diverse problems, and become good independent learners. 

Hence we are faced with a major educational challenge. It is much more difficult to teach effective 
ways of thinking than the memorization of various kinds of factual knowledge. Indeed, these difficulties 
are often all too evident. For example, recent investigations show that many students (even when they 
receive good grades) emerge from basic science courses with gross misconceptions, with prescientific 
notions, and with nominal knowledge that they cannot properly use 

Addressing the challenge . Efforts aiming to transcend the acquisition of factual knowledge must 
identify and teach some important cognitive abilities 5 . (a) For example, much attention must be paid to 
procedural knowledge. Thus it is necessary to teach explicitly the procedural knowledge needed to 
properly interpret concepts, principles, and other declarative knowledge. (Indeed, any factual statement is 
meaningless unless one knows methods specifying how one can determine whether it is true or false.) It is 
also necessary to teach useful methods for analyzing problems and for making the decisions needed to 
construct their solutions, (b) Similarly, one must identify and teach forms of knowledge facilitating 
effective use (e.g., hierarchical knowledge organizations, and complementary qualitative and quantitative 
descriptions in various symbolic representations). 

Successes and limitations of past work . During the last several years one of us (FR) has been engaged 
in work aiming to investigate and teach such cognitive abilities ^“8 This work has shown that some of 
these important abilities can be explicitly taught 9 , that they can be addressed in practical instructional 
materials l0 , and that they lead to substantial improvements of student performance in actual courses 5 . 

However, this work has also revealed some profound educational problems that limit the possible 
efficacy of any instruction, no matter how well it might be designed. The aim of this paper is to point out 
two of these pervasive problems and to describe work showing how they might be overcome. 




2. Pervasive educational problems 

2.1 Lack of individual guidance and feedback 

Learning requirements . Effective learning requires active thinking and practice by the learner. But 
the learning activities must also be of the right kind. For example, hours spent practicing the wrong things 
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can easily be ineffectual or counterproductive, leading to harmful results or to bad habits that are difficult to 
break. 

Instructional efforts should, therefore, be based on a good understanding of the learning processes 
needed to attain proficient performance. However, even such a perfect understanding is not sufficient. One 
must also ensure that students actually engage in these learning processes — otherwise no effective 
learning would occur. (The situation is analogous to that in medicine. Even if one understands perfectly a 
disease and has developed pills guaranteeing its cure, people’s health would not improve if they don’t take 
the pills according to the recommended regimen.) 

A useful way of ensuring that students engage in effective learning processes is to provide them with 
good individual guidance and feedback. Indeed, such individual guidance and feedback are commonly 
provided by coaches or teachers who aim to produce good athletes or fine musical performers. 

Prevailing instructional deficiencies . However, students receive ordinarily very little individual 
guidance and feedback in our educational system where they spend most of their time in large classes or 
studying without supervision. Under these conditions it is nearly impossible to ensure that students engage 
in effective learning processes, even if these processes are well known. Hence there exists a serious 
impediment limiting the possible efficacy achievable even by carefully designed instructional programs. 

For example, textbooks, lectures, and recitation sections may all discuss and demonstrate good 
problem-solving methods. But when students do their homeworks under unsupervised conditions, many 
revert to preexisting habits and try to solve problems by haphazardly grabbing miscellaneous equations. 
Unfortunately, hours of such homework “practice” are likely to do them more harm than good. On the 
other hand, an hour of individual tutorial help may sometimes help a student much more than all the 
lectures, readings, and homework assignments in a course. 

Suggested remedy: Use of computers . How could one provide effective individual guidance and 
feedback to large numbers of students in our educational system? It is clearly impossible to provide every 
student with a good private tutor. But modern computers (and associated electronic media) are now 
increasingly available to many students. Properly designed computer programs could thus help to ensure 
that every student engages in effective learning processes. The guidance and feedback provided by such 
computer programs might well be less good than those ideally available from human tutors. However, they 
might still be much better than what most students currently receive. 

2.2 Deficiencies of basic cognitive functions 

Learning requirements . Good performance of any task requires the following essential cognitive 
functions: (a) Making appropriate decisions, (b) implementing these, and (c) assessing whether 

performance has been satisfactory. 

When tasks are highly familiar (e.g., typing one’s name) these functions are often carried out without 
conscious awareness. But in the case of more complex or unfamiliar tasks, these functions need to be 
carried out quite deliberately. The learning of problem solving or other complex tasks requires, therefore, 
also learning the basic abilities of properly deciding, implementing, and assessing. 

Deficiencies of students 9 cognitive functions . Observations of students suggest that many of their 
difficulties are traceable to deficiencies in these basic cognitive functions. Although students are usually 
most concerned with implementing various actions, their implementations are often faulty. Even more 
frequently, students fail to make appropriate decisions or to assess what they have done. (As Goethe once 
said, “acting is easy, thinking is hard”.) 
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For example, many students don’t perceive the need to assess carefully what they have done, or 
sometimes do not know how to assess effectively their performance. As a result, students often fail to learn 
from their mistakes. Hence their errors or misconceptions may persist for long times. 

Deficiencies of appropriate decision making are even more commonly observed, (a) Such deficiencies 
sometimes cause students not to invoke useful knowledge that they do possess, (b) They often cause a lack 
of reliable sequence control in implementing methods, with the result that steps get omitted or invoked in 
the wrong order. (As a result, one commonly observes among students “procedures that don’t proceed”.) 
(c) Lastly, decision-making deficiencies are particularly fatal in problem solving where judicious decisions 
are of central importance. 

Suggested remedy: A reciprocal-teaching strategy . The preceding deficiencies led us to devise the 
following instructional strategy explicitly designed to improve students’ basic abilities of deciding, 
implementing, and assessing. 

This strategy, illustrated in Figure 1, involves the following two distinct modes of interaction between 
a student S and a tutor T. (The modes are named in accordance with the role played by the student.) 

(1) Implementing mode. In this mode (a) the tutor decides what to do and gives corresponding 
directions, (b) the student implements these, and (c) the tutor assesses and corrects. (In this mode the 
student thus practices implementing while the tutor acts as a coach.) 

(2) Coaching mode. In this mode, the roles are reversed. Thus (a) the student decides what to do and 
gives corresponding directions, (b) the tutor implements these (but may deliberately make mistakes 
reflecting those commonly made by many students), and (c) the student assesses and corrects. (In this 
mode the student practices decision making and assessing, i.e., the student acts somewhat like a coach.) 

During the instruction, the student and tutor repeatedly change roles, alternating between these two 
modes of interaction. This instructional strategy becomes then a “reciprocal teaching” strategy. 

Theoretical advantages of this reciprocal-teaching strategy . Cognitive considerations suggest that 
this strategy should be highly effective for the following reasons: 

(1) The instruction is highly interactive and the student is always kept actively engaged. 

(2) The functions of deciding, implementing, and assessing are made highly explicit. 
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Figure 1. Reciprocal-teaching strategy with alternating modes of 
interaction between a student S and a tutor T. 
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(3) The student practices these functions separately, but always in the full context of an entire 
meaningful task. (Separate practice allows focused attention and reduces cognitive load that may transcend 
learning capacities. But maintaining the entire task context makes apparent all potentially useful cues — 
and provides practice in contexts in which the student will ultimately need to use the cognitive functions to 
be learned.) 

(4) Since the implementing and coaching roles alternate, the tutor repeatedly models good performance 
of a function before the student, in turn, needs to perform this function. 

(5) The tutor repeatedly monitors how well the student performs at any stage — and can then provide 
appropriate feedback and adjust the instruction accordingly. 

(6) The tutor can progressively reduce the level of assistance provided to the student. Thus the student 
can gradually learn more independent and joint performance of the functions of decision making, 
implementing, and assessing. 

Empirical evidence of efficacy , We performed some pilot experiments in which one of us (LAS) 
played the role of tutor in applying this reciprocal- teaching strategy to teach college students some physics 
concepts and principles. The results were quite promising. Not only did the students learn these concepts 
and principles, but they also made more articulate decisions, assessed their work more carefully, and did 
these things more spontaneously. 

The reciprocal-teaching strategy was first introduced by Palincsar and Brown in a very different 
context and with very different aims (to teach reading-comprehension skills to middle-school children). 
The reading strategy taught by them proved highly effective, raising children’s reading-comprehension 
skills from about 30% to 80%. Although their context was quite different, some of the efficacy of their 
reciprocal-teaching reading strategy can also be attributed to the same advantages listed above. 



3. Proposed instructional design 

The preceding considerations suggest that the identified instructional problems could be overcome (and 
effective thinking skills could be more effectively taught) by instructional design incorporating the 
following main guidelines: 

Cognitive analysis , The instruction needs to be based upon an adequate understanding of the thought 
and learning processes leading to good performance. 

Use of the proposed reciprocal-teaching strategy , By embedding instruction in this reciprocal- 
teaching strategy, one can help ensure that the basic cognitive functions needed for good performance (i.e., 
appropriately deciding, implementing, and assessing) are learned and carried out. 

Computers used to implement reciprocal teaching. The role of the tutor T in the proposed reciprocal- 
teaching strategy of Figure 1 can also be played by a properly programmed computer rather than by a 
human tutor. There are two resulting advantages: (a) The reciprocal-teaching strategy can then be more 
readily implemented without the (largely impractical) need to rely on human tutors, (b) Such computers 
can then also provide many students with prompt well-designed individual guidance and feedback of the 
kind commonly lacking in most courses. 

Simple instructional implementations. It is desirable to keep the implementation of the preceding 
instructional guidelines as simple as possible because they could then be more readily realized in practice. 
(In particular, they could then be more widely produced by faculty members or subject-matter experts 
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without much programming expertise.) Indeed, even without resorting to the techniques of artificial 
intelligence, well-designed computer programs could play the role of tutors that provide students with 
much better instruction than many currently receive. To indicate that a computer running such tutorial 
programs may not be endowed with the natural or artificial intelligence of a real tutor, we call it a “PAL” (a 
Personal Assistant for Learning). 



4. Computer tutorials implementing the design 

To explore how the preceding instructional design might actually be implemented in practice, we 
constructed a set of prototype computer tutorials exemplifying the preceding guidelines. The following 
description of these prototypes illustrates more concretely the proposed approach and demonstrates its 
feasibility. 

4.1 Choice of prototypical task domain 

The proper interpretation and application of scientific principles is a demanding task difficult for many 

students. Accordingly we wanted to explore how our approach might be used to teach such principles more 

effectively. In particular, we chose to focus our attention on a fundamentally important principle of 

^ — > — > 

mechanics, Newton’s second law of motion ma = F tot , which relates the acceleration a of any object to the 
total force Ftot acting on it 

This principle is taught in every basic physics course. The thinking skills necessary to interpret and 
apply this principle are typical of those needed in many scientific or engineering fields. Furthermore, 
despite its seeming simplicity, the proper application of Newton’s law causes students many difficulties and 
often becomes a major learning hurdle for them. 

4.2. Cognitive task analysis 

Any principle expresses a relationship between some concepts. The interpretation and application of a 
principle thus requires the following kinds of procedural knowledge: (a) A concept-specification method 
specifying how to describe the related concepts in any specific instance, and (b) a relation-specification 
method specifying how to express the relationship between these concepts in this instance. 

In particular, Newton’s law ma = F tot expresses a relationship between the properties of any “system” 
(the mass of the object of interest and the acceleration describing its motion) and its interactions with other 
objects. (These interactions are described by the total force on the system, i.e., by the vector sum of the 
forces exerted on it by all other objects.) Correspondingly, the application of Newton’s law requires 
procedural knowledge that can be specified by the following two methods: 

(a) Concept-specification method. This method (outlined in Figure 2a) describes the motion and 
interactions of the system in the convenient form of a “system diagram”. It does this by identifying all 
available information about the mass of the system, about its motion (velocity and acceleration), and about 
the forces describing all its interactions. (The method separately identifies long-range interactions, like 
gravity, that are appreciable even if objects are far apart — and contact interactions that are appreciable 
only if objects are so close as to touch each other. Furthermore, it always identifies the interacting object 
before specifying the corresponding force produced by it.) Finally, the method specifies how vectors can 
be usefully decomposed into component vectors along convenient directions. 
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(b) Relation-specification method. This method (outlined in Figure 2b) specifies how Newton’s law is 
used to express the relationship between the motion and interactions of the system. (As indicated in this 
figure, this relationship is most usefully expressed in terms of the numerical components of the related 
quantities along mutually perpendicular directions.) 

The preceding two methods, based on a task analysis, are significantly more specific and detailed than 
those commonly mentioned in textbooks dealing with Newton’s law. (Of course, they also require 
prerequisite knowledge about the properties of velocity and acceleration, about various kinds of forces, and 
about vector components.) 

Our prototype tutorials, described in the following paragraphs, were designed to teach systematic 
application of the preceding two methods to deal flexibly with various situations requiring application of 
Newton’s law. 

4.3 Description of basic tutorials 

Basic implementation tutorials. These tutorials are designed to teach students the basic abilities 
needed to implement application of Newton’s law. Accordingly, PAL (the computer used as Personal 
Assistant for Learning) plays here the role of a coach who makes decisions according to the suggested 
methods — and then assesses and corrects the students’ implementations. 

Each tutorial presents a specific mechanics problem. A simple tutorial, dealing with a single system of 
interest, consists of three sections. These successively aim to describe the motion and interactions of this 
system, to construct the equations expressing Newton’s law, and to explore the qualitative implications of 
these results. 

In the first section of such a tutorial, PAL explicitly displays the concept-specification method of 
Figure 2a (i.e., the method specifying how to construct a diagram describing a system’s motion and 
interactions). PAL gives the student successive directions by following the steps of this method. The 
student implements each of these successive steps and PAL assesses whether the student’s implementation 



(a) Concept specification 



(b) Relation specification 



> Separate diagram of system 

> Mass 

• Motion (> Velocity) (> Acceleration) 

• All forces on system 

• Long-range forces (e.g., gravity) 

> Long-range interacting objects? 
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• Contact forces 
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> Sum of all force components 

• Apply the law along the other direction 

> Choose particular direction 
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> Sum of all force components 




Figure 2. Methods needed for applying Newton’s law. (a) Method for specifying the concepts describing 
a system’s motion and interactions, (b) Method for expressing Newton’s law specifying the relationship 
between a system’s motion and interactions. 
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is correct. If it is not, PAL provides hints or other feedback to help the student correct his or her work. If it 
is correct, PAL indicates the completed step by drawing a corresponding part of the diagram. In this way 
the entire correct diagram, describing the motion and interactions of the system, is progressively 
constructed on the computer screen. (Figure 3 shows a black-and-white illustration of such a multicolored 
screen.) 

In the second section of the tutorial, PAL similarly displays the relation-specification method of 
Figure 2b (i.e., the method specifying how to construct the equations expressing Newton’s law). As before, 
PAL gives the student successive directions following the steps of this method. To implement these steps, 
the student is asked to click on appropriate parts of the diagram to specify corresponding terms in an 
equation. (In this way, PAL tries to maintain a clear connection between the previously constructed 
diagram and the equation expressing Newton’s law.) If the student makes a mistake, PAL again provides 
hints or feedback so that the student can correct it. In this way, the student constructs the relevant algebraic 
equations expressing Newton’s law for the system. (Figure 4 shows an illustration.) 
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Figure 3. A computer screen from an implementation tutorial. (The student, asked to construct a system 
diagram, has just made a mistake in specifying the direction of the acceleration. Hence PAL provides the 
feedback message indicated by the gray arrow.) 
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Figure 4. Another computer screen from an implementation tutorial. (The student, asked to construct the 
equations expressing Newton’s law, has clicked on the force T in the diagram and is now asked to specify 
its component.) 



In the third section of the tutorial, PAL asks questions requiring the student to explore some of the 
qualitative implications of his or her previously constructed equations. (For example, PAL may ask 
whether some quantity of interest is larger or smaller than some other quantity, or how such a quantity 
changes when some other quantity increases or decreases.) In this way, PAL tries to ensure that the student 
does not consider equations as mere symbolic expressions, but can interpret them both quantitatively and 
qualitatively. 

Basic coaching tutorials. In these tutorials, the roles of implementer and coach are reversed. Thus the 
student now acts more like a coach, practicing how to make appropriate decisions and how to assess PAL’s 
implementations of them. 

Each tutorial again presents a specific problem. In the first section of the tutorial, PAL displays, in 
randomized order , the steps of the concept-specification method of Figure 2a. PAL then repeatedly asks 
the student what it (PAL) should do next to describe a particular system. To decide what should be done 
and direct PAL to do so, the student must then select an appropriate step in the proper order. PAL dutifully 
implements this step, but may make mistakes. (Indeed, PAL deliberately makes mistakes reflecting 
common student misconceptions or errors.) After implementing a step, PAL always reminds the student to 
warn it if there are any errors. (If the student does not catch a mistake and asks PAL to proceed, PAL 
expresses misgivings and asks the student to check more carefully.) Whenever the student detects a 
mistake, PAL asks the student to identify the nature of the mistake and to correct it. PAL thus gradually 
manages to draw a correct diagram on the computer screen. (Figure 5 shows an illustration.) 
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Figure 5. A computer screen from a coaching tutorial. (The student has just agreed to wrong contact forces 
drawn by PAL. The message, indicated by the gray arrow, therefore asks the student to check whether PAL 
has indeed correctly implemented the indicated step of the method.) 



The second section of the tutorial is designed similarly, except that it deals with the method of 
Figure 2b for specifying the equations expressing Newton’s law. Thus PAL again implements the student’s 
directions to construct appropriate equations, but may make mistakes. It is then up to the student to identify 
and correct all such mistakes. 

Lastly, in the third section of the tutorial, PAL states some qualitative implications which the student 
again may need to correct. 

Alternating use of these tutorials . In accordance with the reciprocal-teaching strategy, the preceding 
two types of tutorials are used alternately. In one type of tutorial the student can then observe how PAL 
decides, implements, or assesses. In the other type of tutorial, the student himself or herself then practices 
deciding, implementing, or assessing. 

4.4 Description of more advanced tutorials 

More advanced tutorials aim to reduce the guidance to students so that they may learn to work more 
independently under conditions closer to those faced in their courses and future professional lives. 

More advanced implementation tutorials . These tutorials require students to work on paper without 
detailed step-by-step guidance from PAL. They also try to convince students that their own haphazard 
ways of proceeding are often inadequate — and thus to motivate them to adopt the more systematic 
methods suggested by PAL. 
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An advanced implementation tutorial, like one of the previously described basic tutorials, consists of 
three sections (drawing a diagram describing the system, writing the equations expressing Newton’s law, 
and examining their implications). However, the student is immediately presented with the challenge of 
doing an entire such section independently, on paper , without any step-by-step directions. 

PAL then checks the student’s performance by asking a set of questions carefully designed to detect 
deficiencies in the student’s work. (Figure 6 illustrates such a checking question.) If all the student’s 
answers are satisfactory, the student goes on to the next section. Otherwise, PAL warns the student that 
something is wrong, reminds him/her of the systematic method, and may provide a few diagnostic hints. If 
the student is still unable to revise his/her work so as to answer all checking questions correctly, PAL 
guides the student through the section step-by-step according to the suggested method (just as it did in a 
basic tutorial). In this way the student practices the method again and can diagnose the mistakes in his/her 
previous independent work. Finally, the student is presented with his/her previous answers to the checking 
questions so that the student can correct them appropriately. 

More advanced coaching tutorials . In these tutorials PAL solves an entire problem, but occasionally 
pauses either (a) to ask the student to judge the correctness of its actions, or (b) to ask the student for advice 
or help. PAL again makes deliberately some mistakes reflecting commonly encountered errors or 
misconceptions. The student needs then to detect all such mistakes and must tell PAL how to correct them. 




Figure 6. A computer screen from a more advanced implementation tutorial. (The student is here asked 
to check his/her work by entering the equation that he/she obtained by applying Newton’s law.) 
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4.5 Implementation details 

All the preceding tutorials were designed to run on Macintosh computers with color monitors. For the 
sake of simplicity, they were written in “Authorware”, a programming language designed for instructional 
applications and commercially distributed by Macromedia, Inc. 

5. Assessment of efficacy 

We constructed about ten tutorial programs of the preceding kinds and tried them with individual 
students (as well as with some colleagues). These trials led to many pedagogical revisions as well as 
interface refinements. 

Recently we were able to carry out a pilot experiment designed to test these tutorials in a more realistic 
classroom situation. The following paragraphs describe this pilot experiment and its results. 

5.1 Design of the pilot experiment 

Last fall (fall 1996) the introductory physics course for science majors at Carnegie Mellon University 
had an enrollment of about 240 students. A homework assignment dealing with the application of 
Newton’s laws was scheduled to be given to students during the fifth week of that course. We took 
advantage of this opportunity to request permission of the instructor (Professor Gregg Franklin) to have 
some students in the course work on this assignment under somewhat modified conditions. 

Accordingly, we recruited from the course about 75 student volunteers with the offer of providing 
some of them special help with the assignment for that week. (The nature of this special help was not 
specified.) From these volunteers, we selected 45 whom we divided into the following three equal groups 
selected to be of equivalent abilities (by matching their scores on two previous tests given in the course). 

(1) PAL group. This group was to work on an equivalent assignment where most problems were in the 
form of PAL tutorials dealing with problems identical, or very similar, to those in the regular assignment. 
About six students at a time did this by coming, for several 90-minute work sessions, to a special room 
where computers had been set up for their use. 

(2) Tutoring group. This group was to work on the assignment under conditions where they would 
receive individual help from very experienced human tutors (one of us or Professor Jill Larkin). About six 
students at a time did this by coming, for several 90-minute work sessions, to another special room where 
one or two of these tutors were available. 

(3) Control group. This group was to work on the assignment under regular conditions (i.e., with 
access to the ordinary help accessible to all students from the teaching assistants in the course). The 
students in this group were told that we could not accommodate them. However, they were later used by us 
as a comparison group of equally motivated volunteers receiving no special treatments. 

5.2 Implementation of the experiment 

Students varied appreciably in the time they required to complete the assignment. (The average time 
was about 5.5 hours for the PAL group and somewhat less for the Tutoring group.) 

Students in the PAL group were constantly and actively engaged with their computer tutorials and 
required very little outside assistance. Indeed, one person could easily supervise a work session of six or 
seven students, and yet be unoccupied most of the time. By contrast, at least two tutors were required to 
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deal with an equally large work session from the Tutoring group in order to answer the students’ questions 
and keep them from unproductive floundering. 

5.3 Student performance data 

A test, given to all the students in the class at the end of the week after the preceding assignment, 
provided data on how well students had learned to apply Newton’s law. This “Newton test” consisted of 
two problems dealing with application of this law. The tests for all students were graded by the teaching 
assistants in the course. In addition, the tests for the students in our three selected groups were 
independently graded by us (with more detailed attention to the nature of the errors committed by the 
students). 

Test scores . The gross test scores obtained by these two grading methods were nearly the same. They 
are most conveniently summarized by the box plots in Figure 7 which indicate the medians and quartiles for 
the students in the various groups. 

Comparisons of performance on the Newton test The data shown in Figure 7a lead to the following 
conclusions about the comparative performance of the three groups of students: 

(1) The students in the PAL and Tutoring groups (receiving average scores of 76% and 84%) 
performed appreciably better than the students in the Control group (average score of 62%). Indeed, about 
half the students in the Control Group received scores below 65%, while almost none the students in the 
PAL and Tutoring Groups received scores below this amount. 

(2) Compared to the students in the Control group, the students in the PAL and Tutoring groups also 
made appreciably fewer mistakes in identifying forces or applying Newton’s law. 

(3) The students receiving individual human tutoring performed somewhat better than those working 
with the computer tutorials. However, PAL tutorials (even if not improved beyond the present prototypes) 
could be quite effective and could be made readily available to large numbers of students. On the other 
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Figure 7. Test scores of the three groups of students. (The 
box plots indicate medians and quartiles.) 
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hand, it would be practically impossible to provide a significant number of students with many hours of 
expert tutoring help of the kind that the Tutoring group received in this pilot experiment. 

Comparisons with performance on previous tests. It is also of interest to compare the 
students’ performance on this Newton test with their performance on the two prior tests that they had taken 
in the physics course. 

Figure 7b shows the average scores that the students had received on these two prior tests. These score 
distributions are nearly the same for the students in all groups (since the groups had been carefully selected 
to be equivalent). 

Figure 7 indicates that the subsequent Newton test scores of the students in the PAL and Tutoring 
Groups are not very different from those on their prior tests. On the other hand, the subsequent Newton test 
scores of the students in the Control Group are much worse. Indeed, while only a quarter of these students 
received scores less than 65% on the prior tests, about half of these students received scores below this 
amount on the Newton test. 

This result suggests a plausible interpretation. The proper application of Newton’s law is for many 
students a difficult task, requiring proper interpretation of this law as well as integrated application of all 
previously acquired physics knowledge about vectors, motion, and various kinds of forces. The special 
help provided to students in the PAL and Tutoring groups apparently allowed them to face these complex 
task demands without undue difficulties. But, without such special help, many students in the Control 
group could not cope — with the result that their performance substantially deteriorated on a crucially 
important part of the course. 

5.4 Student reactions 

Questionnaire results. Student reactions to the PAL tutorials were very positive. For example, a 
questionnaire, given to students in the PAL group, asked the following question: “How useful were these 
computer tutorials in helping you learn to apply Newton’s laws?”. As indicated in Figure 8, almost all the 
students thought that they were very useful and none rated them less than useful. 

Student interviews. To get more detailed information about students’ reactions, we asked an outside 
person (the director of Carnegie-Mellon’s Teaching Center) to interview a couple of representative students 
from each group. Her report, summarizing her interviews with these students, is revealing. 
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Figure 8. Numbers of students (in the PAL group) giving the indicated 
responses to the question: “ How useful were these computer tutorials in 
helping you learn to apply Newton ‘s laws?” 
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Reporting about her interview with the students in the Tutoring group, she writes: 

“The two students . . . absolutely raved about the impact it had on their understanding of this 
section of the course. Both claimed to have asked numerous questions of the ‘tutors’ and felt 
that they had a much ‘deeper’ and ‘conceptual’ understanding of the principles underlying the 
problem than they had of previous material in the same course.” 

By contrast, reporting about her interview with the students in the PAL group, she writes: 

“The two students who worked on the computer tutorials also raved about their experience, 
although the way they phrased the ‘outcomes’ of the experience was a bit different from those 
students above. One of the students said he now is ‘in the habit of doing certain things/asking 
certain questions’ during the problem solving process, and the other said that what has stuck in 
her head was the process you should go through to solve a problem. They both reiterated (in 
case I didn’t get it, I think) that they learned not only the content, but also the process of solving 
physics problems. ... Like the students above, these two students claimed to understand this 
material much better than previous material, and they attribute this to the computer tutorial.” 

These students (unlike those in the Tutoring Group) seem thus to have acquired from the PAL tutorials 
a more explicit knowledge about relevant thinking processes. (It is noteworthy that the students acquired 
this knowledge merely by working with the computer programs, but were otherwise never told that these 
were designed to teach thinking skills.) 

Finally, the interviewer reports these concluding student comments: 

“The four students involved in the small group tutoring and the computer tutorial all strongly 
recommend continuation of these learning mechanisms. All agreed that they would take 
advantage of them if they were available for the physics course and other courses as well.” 

6. Discussion and implications 

The preceding pages described the use of a computer-implemented reciprocal-teaching strategy to help 
students learn some important cognitive abilities. Our work has, up to now, been merely exploratory. Thus 
the computer tutorials constructed by us have just been prototypes and data about their effectiveness have 
only been obtained from a pilot experiment. Hence there remain many opportunities for further 
investigations and extensions of our proposed instructional approach. 

Basic investigations. It would be useful to investigate in greater detail how the proposed reciprocal- 
teaching strategy promotes effective learning and how it might be improved. This could be done by 
making detailed observations of individual students working with PAL tutorials. Furthermore, these 
tutorials are programmed so as to keep a record of all the actions performed by a student on the computer. 
The analysis of these records provides then also detailed information about the learning processes of 
students working with the tutorials. 

Teaching other thinking abilities. The present PAL tutorials were designed to teach students the 
ability to interpret and apply one particular principle, Newton’s law of mechanics. It would clearly be 
possible to use the same approach to teach also other important principles (e.g., the energy law, the 
momentum law, or other principles outside the domain of mechanics). 

However, the computer-implemented reciprocal-teaching strategy could also be used to teach abilities 
other than the interpretation of principles, (a) For example, it could be even more readily used to teach the 
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interpretation of important concepts that are often difficult for many students (e.g., concepts like 
acceleration, work, potential energy, and others), (b) Furthermore, it would be useful to teach important 
problem-solving abilities (e.g., the abilities to analyze problems and to make the judicious decisions needed 
to construct their solutions). 

Extensions to other domains . The same kinds of conceptual and problem-solving abilities could, of 
course, also be usefully taught in domains outside of our present prototype domain of mechanics. For 
example, similar approaches could be used for teaching electricity and magnetism, for teachng calculus, or 
for teaching in domains outside the natural sciences. 

Teaching general cognitive abilities . Our reciprocal-teaching strategy deals very explicitly with 
decision making, implementing, and assessing. The consistent and repeated use of this strategy to teach a 
variety of specific concepts and principles could then also help students to acquire improved general 
abilities of deciding, implementing, and assessing — abilities that they themselves could then transfer to 
other learning tasks. The teaching of such general learning abilities is clearly an ambitious aim, but some 
of our past work suggests that it might be possible to achieve 14 . 

Practical applications in courses . Computer tutorials of the kind described in the preceding pages 
could be very useful in actual courses. 

(a) They could be used in recitation sections where students might work with the computers singly or 
in pairs. The students, always actively engaged, would then receive good individual guidance and feedback 
while a teaching assistant could answer any questions not satisfactorily handled by the computers. 

(b) They could be used for homeworks that students could do on their own, yet with good guidance and 
feedback. (Access to e-mail would still allow students to get needed additional assistance from an 
instructor.) 

(c) They would also be useful to help students periodically review what they have learned and thus to 
consolidate their knowledge. 

For example, it is fairly clear how we might build on our prototypes to construct further computer 
tutorials that could be very useful in an introductory college physics course. Indeed, such tutorials could 
have great practical utility even if they were no better than those used in our pilot experiment (i.e., even if 
they could merely prevent half the class from falling below test scores of 65%). 

Constructing PAL tutorials for an entire such course would be quite feasible and not excessively 
expensive (especially since the initial cost could later be amortized by repeated and widespread use of the 
tutorials). However, it is doubtful whether we shall have even the minimal resources necessary to produce 
such practically useful computer tutorials. 
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